“Virtual Reality” (VR), “Augmented Reality” (AR), “Mixed Reality” (MR) and others incorporate simulated worlds with or without some portion of the real world of the user. VR creates a virtual world, different than the real world, that is presented to the observer; in AR, simulated object(s) are overlaid (‘augmented’) onto the real world; and in MR the virtual or simulated world is ‘mixed’ with the real world to present a combined world [1]. For this disclosure, VR, AR, and MR, and other simulated content will be called simulated reality (SR), while that of the real world will be called real reality (RR). Some of the companies working on such SR devices include Facebook Oculus, Microsoft Hololens, Magic Leap, Vuzix, Google, Samsung, etc. Many of these, if not all, devices are head-mounted displays (HMD) with a history of designs and applications that include the military, commercial, and consumer space [2,3].
VR devices, such as those by Samsung or Oculus, typically display two stereoscopic images (dynamic or static), one image to each eye, at a time. Most, if not all, VR devices do not allow users to see through the device as they are not transparent. However, some use detection schemes to then add on some level of simulated images of the environment of the user. Also, lenses are usually placed between the images and eyes to allow the eyes to focus on the images as if they appear from a far distance (“infinity” in optical terms) [4]. Google glasses and similar devices incorporate AR by displaying graphics on to the glasses worn by the user [3]. Magic Leap and Microsoft Hololens use MR, where users can interact with the simulated content that is incorporated in with the real world, with the real world (or portions of it) still being visible. Most, if not all, AR or MR devices, and the likes, incorporate the reality of the user through transparent devices, so the user can see the real environment directly.
The concept of a “light field” is important in describing light for image capturing and display purposes. Light is an electromagnetic wave that oscillates in time, or a combination of multiple waves. At a given point in time, each ‘component’ of light can be described by its color (frequency), amount/strength (irradiance), position, direction, and phase. Since human eyes cannot observe the ‘phase’ of visible light waves, ignoring this allows us to describe light as rays and simplifies the calculation of light propagation [5]. In particular, for a given time, a light ray can be described by its position and direction, together with its color and irradiance. This is called a ‘light field’ and collection of light fields can allow for 3D images to be captured and displayed [6]. To simplify discussion in this disclosure, we will assume that the color and irradiance of a light field is known (which can be measured with detectors such as those in cameras), and simply use its direction and position to describe the light field.
Some research and development of three-dimensional (3D) light capturing and display have looked into capturing and using light fields, together with computational imaging and post-processing calculations and effects [7]. Light field ideas predate many of the recent research and developments in this field, but with increased computational power and engineering, the quality and capabilities have become more practical than before. Even groups working in the VR, AR, or MR fields have begun to investigate using light fields, including Magic Leap [1].